[2.8] Warn on deploy prepare storage rewrites#4530
[2.8] Warn on deploy prepare storage rewrites#4530YuanTingHsieh merged 11 commits intoNVIDIA:2.8from
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves nvflare deploy prepare for containerized runtimes by relocating server-side job/snapshot storage into the mounted workspace (to align Docker with existing K8s behavior) and by surfacing warnings when prepare-time component rewrites would otherwise silently discard operator configuration.
Changes:
- Relocate Docker server snapshot/job storage paths into the mounted workspace during deploy prepare (matching K8s behavior).
- Share/extend the server storage relocation helper and add warnings for non-relocatable snapshot persistor shapes.
- Warn when deploy prepare replaces/removes existing resource manager/consumer and launcher components/config during runtime preparation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
nvflare/tool/deploy/deploy_commands.py |
Adds Docker server storage relocation, introduces warning helpers for discarded component config, and warns on unexpected snapshot persistor shapes. |
tests/unit_test/tool/deploy/deploy_commands_test.py |
Adds unit tests for Docker server storage relocation and for warning/silence behavior around snapshot persistor shape and component replacement. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Greptile SummaryThis PR relocates Docker server job-history and snapshot storage into the mounted workspace (matching the existing K8s behaviour), and adds per-component warnings when
Confidence Score: 5/5Safe to merge; all code paths are well-guarded and covered by new tests. The change is narrowly scoped: it adds a read-then-warn pass before component replacement and wires up Docker server storage relocation that was already working for K8s. The warn-before-mutate ordering is correct, the builtins sets are complete for the current launcher catalogue, and the new tests verify both the happy path (relocation succeeds, no spurious warnings) and the error paths (malformed persistor, custom config discarded). The only edge case noted — a JSON-null snapshot_persistor emitting a misleading warning — has no impact on the core relocation or patching logic. No files require special attention. Important Files Changed
Sequence DiagramsequenceDiagram
participant CLI as CLI / prepare_deployment
participant PD as _prepare_docker / _prepare_k8s
participant PR as _patch_resources
participant WRC as _warn_for_replaced_components
participant RSS as _relocate_server_storage_to_workspace
participant FS as resources.json.default
CLI->>PD: prepare (kit_info, config)
PD->>PR: (kit_dir, launcher_id, launcher_path, launcher_args)
PR->>FS: read resources
PR->>WRC: components, launcher_id, launcher_path
WRC-->>PR: emit Warning: (if custom resource_manager / resource_consumer / launcher found)
PR->>FS: write patched resources (passthrough RM + new launcher)
alt "role == ROLE_SERVER"
PD->>RSS: (kit_dir, workspace_mount_path)
RSS->>FS: read patched resources
alt snapshot_persistor present
RSS->>RSS: update snapshot_persistor.args.storage.args.root_dir
note right of RSS: emit Warning if KeyError / TypeError
end
RSS->>RSS: update job_manager.args.uri_root
RSS->>FS: write relocated resources
end
PD-->>CLI: result dict
Reviews (7): Last reviewed commit: "Merge branch '2.8' into codex/deploy-pre..." | Re-trigger Greptile |
…-storage-warnings # Conflicts: # tests/unit_test/tool/deploy/deploy_commands_test.py
…nings' into codex/deploy-prepare-storage-warnings
…-storage-warnings # Conflicts: # tests/unit_test/tool/deploy/deploy_commands_test.py
same as #4530 ## What changed - Port the deploy prepare storage relocation and observability warnings from the 2.8 PR to main. - Relocate Docker server job history and snapshot storage into the mounted workspace, matching K8s server prepare behavior. - Warn when deploy prepare replaces or removes existing resource manager, resource consumer, or launcher configuration. - Warn when a present snapshot_persistor cannot be relocated because it does not expose snapshot_persistor.args.storage.args.root_dir. ## Why Containerized deploy prepare rewrites runtime components and relocates server storage. Without these warnings and Docker storage relocation, operator customization can be silently dropped and server state can remain on container-local storage. --------- Co-authored-by: Peter Cnudde <pcnudde@nvidia.com>
What changed
snapshot_persistorcannot be relocated because it does not exposesnapshot_persistor.args.storage.args.root_dir; absentsnapshot_persistorremains silent.Why
Containerized deploy prepare rewrites runtime components and, for K8s, relocates server storage to durable workspace storage. Before this change, Docker server prepare left default job/snapshot stores under container-local
/tmp/nvflare/..., and both component rewrites and malformed snapshot persistor layouts could silently drop operator intent.Validation
python3 -m py_compile nvflare/tool/deploy/deploy_commands.py tests/unit_test/tool/deploy/deploy_commands_test.py